USPS EXPRESS MAIL 
El 759 601 256 US 
HAY 09 2001 



Docket -^W 97 



l 



Image recorder and image recording method, in particular for recording 
objects or scenes in three dimensions 



BACKGROUND 
Field of the Invention 

The present invention relates to an image recorder with a detector unit 
10 comprising a multiplicity of photosensitive pixel elements for generating 
signals containing information about individual image points, and a signal 
output for outputting image information to which the signals of the pixel 
elements are fed. 

The invention furthermore relates to an image recording method in which 
15 signals containing information about individual image points are generated 
by means of photosensitive pixel elements, and the signals are then made 
available as image information. 

The image recorder and the image recording method are particularly 
suitable for recording objects or scenes in three dimensions. 



Description of the relat e d Techno l ogy 

Image recorders with a multiplicity of photodetectors are used in many fields 
25 of technology, for example in remote detection, for monitoring rooms and in 
the consumer goods industry. Three-dimensional measuring systems which 
are suitable for recording 3D range images serve, for example, for optically 
detecting the shape of objects or surfaces, whereby information about the 
distance of individual image points is also contained in the recorded image. 
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An area of application of great practical importance is the fast detection of 
the 3D shape of any surfaces at distances ranging from 20 cm to 50 meters. 
High speed recording and processing of image information is particularly 
necessary for navigation, motion control and the fast 3D surveying of large 
5 contours. 

Known approaches to recording three-dimensional range images are, for 
example, radar and laser radar systems. However, they are complex to 
manufacture and incur high costs. Another possibility is offered by silicon 
10 image recorders, which are used in a range image camera. This admittedly 
achieves a high image resolution according to VGA standard with, for 
example, 640 x 480 pixels, however the analysis of the recorded scenes 
requires considerable time, even with high computing performances. 

15 Photonic mixer devices (PMD) are known as alternative silicon image sensors 
for 3D range images. Such photonic mixer devices or photonic mixer 
detectors are described in DE 197 04 496 Al. A pixel of a photonic mixer 
element has at least two light-sensitive modulation photogates and assigned 
accumulation gates in order to record the phase and/or amplitude 

20 information of an electromagnetic wave. Gate voltages are applied to the 
modulation photogates, whereas a direct voltage is applied to the 
accumulation gates, whereby the charge carriers which are generated by an 
incident electromagnetic wave in the spatial charging zone of the modulation 
photogate are exposed to the potential gradient of a drift field, as a function 

25 of the polarity of the gate voltages, and drift to the corresponding 
accumulation gate. 

R. Schwarte et. al. also described photonic mixer detectors or PMDs during 
a lecture entitled "Fast and simple optical shape detection with a new type of 
30 correlation photodetector array" to the DGZfP-GMA Technical Meeting in 

Langen on April 28/29, 1997. In the associated lecture paper, among other 
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items, a ID PMD laser radar with a PMD pixel is shown, in which the 
distance of a target point in a 3D scene is determined via a phase evaluation 
of an electromagnetic wave and/or via the phase runtime. 

5 Nevertheless, the problem with the known devices and methods for three- 
dimensional image recording is that a very great computation effort is 
required, as well as expensive electronics, particularly for recording a 
complex scene. This impairs the fast acquisition of information because of 
the time required. Fast three-dimensional pattern recognition is not possible 
10 or only possible with severe limitations, or with a disproportionate effort and 
high costs. 

Summary of the invention 

15 The object of the present invention is to provide an image recorder and an 
image recording method with which complex scenes can be recorded, even 
in three dimensions, with increased speed without incurring high 
expenditure. 

20 According to the first aspect of the present invention, this object is achieved 
such that the pixel elements contributing to the image information are 
arranged irregularly, and/or their arrangement is adapted or can be adapted 
to the expected positions of characteristic image elements. 
According to the second aspect of the present invention, this object is 

25 achieved such that the signals made available correspond to image points, 
the arrangement of which is adapted to the expected positions of 
characteristic image elements, and/or the image points are arranged 
irregularly. 

30 The image recorder according to the invention is particularly suitable for 
recording objects or scenes in three dimensions, and comprises a detector 
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unit which has a multiplicity of photosensitive pixel elements for generating 
signals containing information about individual image points, and a signal 
output, to which the signals of the pixel elements are fed, for outputting the 
image information, whereby the pixel elements contributing to the image 
5 information are arranged irregularly, and/or their arrangement is adapted or 
is adaptable to the expected positions of characteristic image elements. 

The image recorder according to the invention or 3-D image sensor enables 
a high processing speed and in particular fast recognition of 3D patterns to 
10 be achieved. Particularly cost-effective 3D image sensors can be produced 
by using a minimized pixel count which is, for example, restricted to a 
defined arrangement. The pixel arrangement is thus, for example, adapted to 
a scene which is to be recorded so that the image of the characteristic 
features of the scene can be recorded with just a few pixels. 

L5 

It is advantageous if the pixel elements can be addressed directly. The pixel 
elements can also be addressed sequentially or optionally. In the case where 
the individual pixels are addressed sequentially, e.g. via a shift register, the 
entire image information can be made available as a sequential data stream. 

20 

The addressing of the pixel elements is, for example, discretionary or it may 
be made in a predefined sequence. 

For example, the pixel elements contributing to the image information may 
25 be arranged in a V or cruciform shape. However, they may also be arranged 
in a line, for example with irregular spacings between each other. In 
particular, in this way a higher pixel density can be provided in so-called 
regions of interest (ROI) than in less interesting areas. 

30 It is advantageous if the pixel elements are programmable so that, from a 
multiplicity of pixel elements, selected pixel elements may be activated 
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and/or deactivated. In this case, the detector not only contains a smaller 
number of pixel elements, but also only specific pixel elements in a line or 
line arrangement are activated in order to record the essential image 
information. The image processing speed is increased by dispensing with the 
5 recording and processing of unimportant image information. 

Preferably, programmable elements are provided for selecting the pixel 
elements. Advantageously, suitable switches are used to bridge individual 
pixels and/or to interrupt address lines. For example, EPROM chips may be 
10 provided, or even fusible electrical links with which individual pixel elements 
may be permanently activated or deactivated. 

Preferably, the image recorder comprises a line decoder and/or a column 
decoder for addressing the programmable elements. In particular, a shift 
15 register serves to read out the pixel elements contributing to the image 
information or selected pixel elements. 

Preferably, one or a plurality of video lines are provided which serve to 
output the signals of the pixel elements. In the image recorder according to 
20 the invention, there are advantageously photonic mixer devices or photonic 
mixer detectors with which three-dimensional range images can be 
recorded. 

in the image recording method according to the invention, in particular for 
25 recording objects or scenes in three dimensions, by means of photosensitive 
pixel elements, signals are generated containing information about individual 
image points, and these signals are then made available as image 
information, whereby the signals made available correspond to image points, 
the arrangement of which is adapted to the expected positions of 
30 characteristic image elements, or the image points are arranged irregularly. 
Processing speeds can be increased with the method, so that measuring 
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tasks under known scene conditions can be fulfilled with a minimum number 
of detector elements. The image processing tasks are thus made 
considerably easier. In particular, the pixels are adapted to the expected 
scene and/or to the image processing algorithms used. 

5 

Permanently arranging a reduced number of pixel elements, for example 
irregularly or in various geometric shapes, also enables substantial costs to 
be saved. The quantities of data for further image processing are kept as 
small as possible. This is done, for example, by using existing knowledge 
10 about the scene to be analyzed in order to configure detector blocks 
optimally. 

In the image recording method, the pixel elements are advantageously 
addressed sequentially or optionally. The sequence in which the pixel 
15 elements are addressed may also be specifically selected for the application. 

For example, the pixel elements may be addressed by line jump, or 
meandering line-by-line. 

20 Advantageously, the pixel elements are programmed in order to activate 
and/or deactivate selected pixel elements. In this manner, pixel elements 
can, for example, be activated or selected specifically to meet different 
requirements. 

25 Advantageously, the pixel elements are dynamically reprogrammed during 
the recording. This makes it possible to adapt to changing scenes during the 
recording, whereby the data processing speed is nevertheless increased 
without losing crucial information. 
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Brief Descri pti on of the Figure s 

The invention is described in the following with the aid of examples and the 
figures. They show: 

5 

Fig. 1 a PMD detector line with an irregular pixel arrangement as a first 
preferred embodiment of the invention; 

Fig. 2 a PMD detector array with a V-shaped region of interest (ROI) as a 
10 further embodiment of the invention; 

Fig. 3 a PMD detector array with a cruciform region of interest 
corresponding to yet another embodiment of the invention; 

15 Fig. 4 a PMD array with optional addressing; 

Fig. 5 a PMD array with sequential addressing; 

Fig. 6 a PMD detector line with pixel programming corresponding to another 
20 preferred embodiment of the invention; 

Fig. 7 the architecture of a PMD array with programmable pixels; 

Fig. 8 an array read out in the line jump method, and 

25 

Fig. 9 an array read out which proceeds meandering line-by-line. 
Description of the preferred embodiments 



30 A detector line 10 is shown diagrammatically in figure 1. It contains pixel 
elements 11 and pixels or image recording points which are arranged 
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irregularly. In which some pixel elements lie directly adjacent, whereas there 
is a physical distance between other pixel elements. The distances between 
the pixel elements 11 vary in size so that so-called regions of interest 12 are 
formed which have a higher pixel density than less interesting areas. 
5 Whereby the arrangement of the pixel elements 11 with the regions of 

interest 12 is adapted to the scene which is expected to be recorded by the 
image recorder. 

In the image recorder according to the invention, the arrangement of the 
10 pixel elements 11 and the detector line 10 is freely selectable in the focal 
plane, so that the image recorder is adapted to the measuring task of the 
measuring system in order to record three-dimensional objects or images or 
scenes. 

15 The detector line 10 may, for example, be used in production or 

manufacturing to determine the location of work pieces. With specified 
locations, the image recording and image processing are restricted to those 
areas of the image in which work pieces can be found, whereas the other 
areas do not take up any resources or computing time. 

20 

A detector line 20 is shown in figure 2. In which all the pixel elements 21 are 
arranged in a V-shape so that they form a V-shaped region of interest (ROI). 
The V-shaped preferred lines of the pixel elements 21 as ROI are particularly 
suitable for pre-crash sensing in vehicle technology. In this case, a V-shaped 
25 image section projecting forwards from the vehicle is recorded which is 

processed at such a high speed that hazardous situations are automatically 
detected in good time, and appropriate automatic functions such as warning 
or steering functions can be executed. 

30 Figure 3 shows a detector arrangement 30 with pixel elements 31 arranged 
cruciformly. This array arrangement has a horizontal and a vertical line as 
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region of interest or ROI. Such an image recorder is, for example, arranged 
in a vehicle in order to detect obstacles on streets, preferably with the 
horizontal line of pixel elements 31. Vertical obstacles, for example limited 
clearance heights in garage entrances or tunnels, are detected by the 
5 vertical line as ROI. Here again, the number and arrangement of the pixel 
elements 31 is adapted to the measuring task and/or to the expected scene 
so that obstacles can be quickly detected with cost-effective image 
recorders. Tracking tasks and tracking measures can also be realized quickly 
and cost-effectively with the image recorder according to the invention. 

10 

The individual pixels or pixel elements 11, 21, 31 are addressed, for 
example, directly with individual signal outputs. 

Figure 4 shows the detector arrangement 20 with V-shaped ROI and optional 
15 addressing. The pixel elements 21 are individually or singly controlled by a 
matrix of horizontal address wires 32 and vertical address wires 33. A pixel 
element output wire 34 runs from each pixel element 21 to a video signal 
. output 35. 

A column address unit 36 is linked to the individual pixel elements 21 by the 
20 vertical addressing wires 33. A line address unit 37 is linked to the 
individual pixel elements 21 by the horizontal address wires 32. 

In the detector arrangement 20 with a V-shaped ROI shown in figure 5, the 
pixel elements 21 are addressed sequentially with the aid of a shift register. 

25 Whereby the sequence of pixels is, for example, specifically defined for the 
application. Here, only a column addressing unit 36 is provided, which is 
linked by an address wire 33 to each of the pixel elements 21. in operation, 
the generated pixel signals are output from the 3D range image recorder on 
one or a plurality of video lines. The video signal output 35 is shown 

30 diagrammatically in the figure. 
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The pixel elements 11, 21, 31 are photonic mixer devices as described in 
detail in DE 197 04 496 Al, which was mentioned at the beginning, to the 
content of which express reference is made here. In respect of the concrete 
embodiment of the pixel elements 11, 21, 31, reference is made once again 
5 to the paper mentioned in association with the lecture referred to at the 
beginning. The pixel elements 21 of figure 4 are addressed by multiplexers. 

Figure 6 shows a detector line 40 with programmable PMD pixel elements 
41. A shift register with shift register stages 43 serves as an addressing unit 
10 for the individual pixels or pixel elements 41. 

The flip-flop stages of the shift register can be individually bridged by 
programmable elements 42, of which one is contained in each pixel element 
41, or is assigned or coupled to it. 

15 The programmable elements 42 each contain a switch SI which in the 

closed state bridges the associated flip-flop stage or shift register stage 43. 
The associated pixel, which is assigned to the shift register stage 43, is thus 
not addressed in this state of switch SI and is skipped in the serial signal 
read out. 

20 

A further switch S2 in the programmable element 42 makes the contact to 
the associated pixel element 41. With switch S2 in the open state, the pixel 
element 41 is thus additionally separated from the shift register stage 43. 

25 The pixel element 41 associated with a programmable element 42 is thus 
deactivated with switch SI closed and switch S2 open, and activated with 
switch SI open and switch S2 closed. In this manner, any selection of pixels 
or pixel elements 41 can be made from the entire detector line 40. The 
programmable element 42 can be addressed in a simple manner via a 

30 decoder block. 
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Such switches can be realized, for example in CMOS technology, with the aid 
of a few transistors. The switch setting is programmed, i.e. the potentials at 
the gates of the switch transistors are set, via, for example, EPROM chips, 
which are also arranged on the image recorder. However, so-called fusible 
5 links, which create a permanent switch setting during the programming, may 
also be used. Both options may be implemented in the programmable 
elements 42. 

When programming the programmable elements 42, one or a plurality of 
10 pixel elements 41 are excluded from the addressing, in that, as mentioned 
above, the input D and the output Q of the shift register stage 43 are bridged 
by the switch SI. Additionally, the address line leading from the connection 
Q of the shift register stage 43 to the pixel element 41 is interrupted by the 
programming via the above-mentioned switch S2. 

15 

A decoder unit 45 is linked to each of the programmable elements 42 by 
address wires 46 so that they can be addressed. 

When EEPROM chips are used for realizing the image recorder or detector 
20 block, the selection of the pixels or pixel elements 41 can be dynamically 
changed during detector operation. This enables the image recorder to be 
dynamically adapted to varying scene conditions. 

Figure 7 shows a two-dimensional array arrangement of pixel elements 41 
25 which can be selected, activated or deactivated by the programmable 

elements 42. Whereby a horizontal shift register 48 is assigned to each pixel 
line 47 of the array. A line decoder 50 and a column decoder 51 serve to 
address the programmable elements 42. After programming, only the 
selected pixels are read out in a serial data stream to one or a plurality of 
30 video lines. 
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Figure 8 shows the array read out in the line jump method diagrammatically. 
In which the read-out always takes place in the same direction. In the array 
of pixel elements shown here, selected pixel elements 52 are activated by 
programming to form a V-shaped arrangement of active pixels. 

5 

Figure 9 shows an array read out which proceeds line-by-line in a 
meandering manner. In which the read-out direction alternates from line to 
line, so that after running through a shift register 48, the following shift 
register is run through in the opposite direction. Here again, selected pixel 
10 elements 52 are activated according to a preset region of interest ROI. 

In figures 8 and 9, the selected pixel elements 52 are depicted with dark 
shading so that the V-shaped region of interest on the image recorder is 
recognizable. The number of video lines can be adapted to meet the 
15 requirements of the application. As.a rule, the video line is implemented as a 
two-wire system for the differential read out of the PMD signals. 

To summarize, the invention creates a 3D range image recorder, preferably 
according to the PMD principle, in which a detector line or a detector array 

20 can be equipped with programmable pixels, and in which the application- 
specific pixel selection is preferably made by programming and the pixels 
can be optionally programmed. Furthermore, the pixels can be dynamically 
reprogrammed during detector operation, and a two-dimensional array of 
pixel elements can be addressed line-by-line by shift registers. The 

25 addressing may take place by line jump or meandering line-by-line, and the 
detector signals are output from the 3D range image recorder via one or a 
plurality of video lines. 

The detector line or the detector array may have an irregular, application- 
30 specific PMD pixel arrangement according to the requirements of the 
application concerned. The pixels may be addressed sequentially or 
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optionally, whereby with sequential addressing the sequence of the pixels 
can be selected specifically for the application. 

In all, the invention enables 3D range images to be recorded with the aid of a 
5 silicon image recorder, in which the positions of the image points in the focal 
plane are adapted to the positions of characteristic features within the scene 
to be recorded. In this manner, a complex scene with known features can be 
analyzed in real-time with a low number of image points. The image 
recorders required for this are simple and cost-effective. Instead of arranging 

10 the pixels in a detector line or detector array regularly and at a high overall 
density, in the present invention only a low number of pixel elements are 
provided for recording the characteristic features of the scene, in which the 
few pixel elements are either permanently arranged or, from a large number 
of pixels, individual pixels can be activated or deactivated by appropriate 

15 programming. A particularly fast sequential image read-out process takes 
place by means of shift registers. Dynamic reprogramming of the pixels 
during detector operation enables flexible selection of the pixels. Above all, 
the invention enables fast 3D pattern recognition by means of a pixel 
arrangement adapted to the scene. 

20 

With the range image recorder according to the invention, the head positions 
of the occupants of automotive vehicles can, for example, be determined in 
order to individually adapt the quantity of gas to be emitted into an airbag to 
the head position, and so reduce the risk of injury. It is also possible to have 
25 an out-of-position sensor system in which a camera monitors the driver and 
passenger to determine their positions. 

A further possible application is the pre-crash sensor system, in which the 
range image recorder very quickly automatically detects whether persons or 
30 objects are in the vicinity of the vehicle in order to then intervene in the 
vehicle functions, for example, by braking or triggering a warning function. 
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The range image recorder according to the invention and the method for 
obstacle detection are particularly suitable, for example, in stop and go 
traffic in which the preceding vehicle is monitored and braking is initiated 
5 upon reaching a minimum distance which is, for example, a function of the 
prevailing driving state. 

The present invention may also be part of an interior monitoring system, for 
example, in order to detect a change of state in the vehicle and trigger an 
10 alarm. In this way, the interior of the vehicle can be monitored in a cost- 
effective manner with only a few image elements, in particular for 
safeguarding against theft. Furthermore, many different applications are 
possible, for example in automation, manufacturing technology and in other 
areas. 
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